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elements and summarizes related research used to further refine and extend 
the program. (Contains 120 references and 5 tables.) (Author/YDS) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Tom Marcinkowski 



An Issue and Action Instruction Program 



O 

oo 



\o 



Q 



W 



AN OVERVIEW OF AN ISSUE AND ACTION INSTRUCTION 
PROGRAM FOR STEWARDSHIP EDUCATION 

Tom Marcinkowski 
Florida Technical Institute 



Abstract - The purpose of this paper is to summarize an issue instruction environmental education program devel- 
oped by Harold Tlungerford and his associates over the past three decades. The primary focus of the issue instruc- 
tion program is on the development and application of investigation and evaluation skills to environmental problems 
and issues. Program materials take a structured approach to the development of these skills and encourage the 
teacher to become a guide and facilitator during the skill application. The paper documents the development of 
various program elements and summarizes related research used to further refine and extend the program. 



Preface 



Early in the development of this paper and docu- 
ment, I expressed some reservations about the term and 
concept "best practices." I would like to briefly summa- 
rize my concerns here as a short preface to this paper. 

To begin, it should be acknowledged that "best 
practice" is one of several terms that now' appear in the 
literature and in informal use. Several other terms in use 
include "standard practice," "sound practice," and 
"common practice." Holsman (2001) refers to common 
practices as those that are commonly used in a given 
field. Common practices are usually practical (i.e., they 
have worked in practice over time), and have some tra- 
ditions associated with their use. Further, it is often as- 
sumed there is some type of research base to them, 
whether or not that is true. Professionals (or fields) can 
get into trouble with common practices when tradition 
overrides the absence of research, or worse, when re- 
search runs contrary to assumptions related to those 
practices. For example, in EE, many have assumed that 
the Raths-Harmon-Simon approach to values clarifica- 
tion is supported by research, despite a caretul review' 
of this body of the research that clearly indicates other- 
wise (Lemming, 1985). 
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Next, I’d suggest that the term "best practices" can- 
not be easily applied to aquatic resource education 
(ARE) or environmental education (EE). First, from a 
research and evaluation perspective, the evidence base 
in the fields of ARE and EE is limited in its ability to 
support strong claims about the value of many of our 
practices (and programs). In some cases this research 
has not been done and. in cases where it does exist, it 
has not been fully synthesized. This makes it difficult 
for professionals in ARE and EE to justify or defend 
their claims about what works (best) using anything 
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other than what Michael Quinn Patton refers to argu- 
ments of intrinsic value. In the current atmosphere of 
accountability, intrinsic arguments carry far less weight 
than do data based ones. 

Second, from a practical perspective, teachers and 
other educators know that practices (or programs) that 
work best for one population of learners in one place at 
one point in time may not be best for another population 
of learners in another place and/or at another time. This 
view is embedded in much of the qualitative research in 
education (e.g., field and case studies). Even in quanti- 
tative research, researchers concede this point when 
they indicate that some practices are more robust or 
generalizable than others (i.e., there is a narrower or 
wider range of populations and settings in which prac- 
tices are determined to be useful). It takes a substantial 
amount of research, development, and evaluation work 
on a given practice to support claims that it has, in fact, 
been found to be useful for a variety of populations in a 
variety of setting (e.g., the extensive body of work and 
research on cooperative learning). Few practices (or 
programs) in EE have received this kind of systematic 
attention. For this reason, it is noteworthy that the pro- 
fessional educators associated with the program fea- 
tured in this paper have worked hard to address these 
needs and concerns for more than 25 years. 

Further, the terms standard practice and sound 
practice have been used in a variety of ways. For exam- 
ple, standard practice can refer to the standardization of 
a given practice for the sake of consistency from place- 
to-place and time-to-time (i.e., from a technical 
perspective). In this sense, a practice may or may not 
have either an evaluation or a research base that support 
its use. To overcome this weakness, Holsman (2001) 
defines standard practice in a narrower sense: "Standard 
practices are hereby defined as ’those teaching or educa- 
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tional strategies which appear to yield desired outcomes 
if applied under a certain set of conditions, and with age 
appropriate audiences as demonstrated by research and 
evaluation.” Similarly, for the purposes of this paper, I 
will define sound practice as meeting two criteria im- 
plicit in this discussion: (a) from a practical perspective, 
a program may be considered sound if its design, deliv- 
ery, and evaluation work well to help learners accom- 
plish specified goals and objectives; and (b) from a re- 
search and evaluation perspective, a program may be 
considered sound if there is a well developed body of 
evidence to document and support (a). Thus, Holsman’s 
definition of standard practice and this definition of 
sound practice appear to address similar criteria and 
concerns. 

Given the apparent disadvantages of "best prac- 
tices,” 1 will close by indicating a preference for either 
’’standard practices” or "sound practices" as defined 
above. Both terms, as defined, suggest that a practice 
(or program) has been clearly defined, refined through 
repeated delivery, and supported by a substantial body 
of evidence. Still, both terms leave unanswered impor- 
tant questions regarding the extent to which any prac- 
tice (or program) can be generalized to new' populations 
and settings; these questions should always be open to 
scrutiny. 1 believe that the ideas expressed in this pref- 
ace are important for those involved in boating, fishing 
and stewardship education, and leave it to readers to 
consider them as they read this paper and document. 

Model Used to Organize and Illustrate This Paper 

The purpose of this paper is to summarize a par- 
ticular Environmental Education (EE) program. Unfor- 
tunately, this purpose and the format of this document 
do not permit me to adequately present this program. 
The best way to do that would be to arrange for readers 
to become involved in or to directly observe training 
sessions and classroom implementation, or perhaps to 
view any number of taped workshop and classroom 
segments. Consequently, other approaches must be used 
to summarize this program. The two that are most 
common in the literature and that will be used through- 
out tliis paper are narrative summaries and graphic or- 
ganizers. An attempt will be made to enrich these with 
the use of quotes and selected samples from program- 
related materials, studies, and articles. 

The graphic organizer or model that I have used to 
organize this paper and will use to illustrate this pro- 
gram must be described in some detail (Figure 1). Dr. 
R. Ben Peyton of the Department of Fisheries and Wild- 
life at Michigan State University presented an earlier 
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version of this model in his w'ork with Aquatic Re- 
source Education (ARE) programs in Northeast and 
Mid-Atlantic states. He used that version of this model 
to help State ARE Coordinators explore the distinctions 
between "Outcomes" and "Benefits," and the common 
tendency for program administrators to select ARE pro- 
gram "Tools" before determining ARE program "Pur- 
poses." Since 1997, I have evolved the version of the 
model that now appears in Figure 1. 

A number of factors have contributed to the 
graphic model that appears in Figure 1, notably ad- 
vances in the field of program evaluation: (a) where 
there is an increasing use of the terms "Outputs," "Out- 
comes," and "Impacts” to distinguish among different 
kinds of program results (Rossi, Freeman and Lipsey 

1999) ; and (b) where this is increasing interest in and 
attention to logic modeling to graphically depict key as- 
sumptions about and features of programs that have 
evolved a theory and evidence base (McLaughlin and 
Jordan 1999; Rossi. Freeman and Lipsey 1999; Rogers 

2000) . A brief description of these should help readers 
better understand the nature of the graphic model in 
Figure 1. 

Outputs, Outcomes , and Impacts 

Allow me to describe how r the three terms from (a) 
above are used with respect to the model in Figure 1. To 
begin, Rossi, et al. (1999) describe the difference be- 
tween outputs and outcomes. 

in particular, performance measurement 
schemes distinguish program outcomes from 
program outputs. Program outputs are the 
products or services delivered to program par- 
ticipants or other such activities view'ed as part 
of the program’s contribution to society. Meas- 
ures of output, for example, would relate to 
such things as the number of clients served, the 
number of service units provided, costs per 
service unit, the quality of services provided, 
the narure and volume of advocacy or promo- 
tional efforts made by the program, and so 
forth, (pp. 201-202, underlining added) 

Thus, outputs refer to: (a) number counts associated witli 
the delivery of a program, such as those traditionally re- 
quested by federal and state agencies; and (b) the quality 
of the delivered program, as is traditionally determined 
through formal and informal feedback mechanisms (e.g., 
satisfaction surveys). 

In contrast, Rossi, ct al. (1999, p. 202) describe out- 
comes as the more immediate results of program activity 
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Figure 1 - A General Logic Model for Resource Education Programs 
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ties, described in tenns of the effects of these activities 
on program participants (e.g., increased awareness and 
skill). For educational programs such as ARE and EE 
programs, outcomes refers to the more immediate learn- 
ing outcomes in the cognitive, affective, psychomotor, 
and/or behavioral domains that may be observed at the 
end of a program (e.g., gains related to any of the Tbilisi 
Objectives; Unesco, 1977). The measurement of out- 
comes is typically referred to as the assessment of learn- 
ing or of learning outcomes (Jozzi, Laveault, and Mar- 
cinkowski 1990). 

Lastly, Rossi, et al. (1999), describe impacts as part 
of their discussion of program impact theory. 

The central premise of any social [or envi- 
ronmental] program is that the service it de- 
livers to the target population induces some 
change that improves social [or environ- 
mental] conditions. The program impact 
theory is the set of assumptions embodied in 
the program about how its services actuate 
or facilitate the intended change. Program 
impact theory, therefore, is causal theory: It 
describes a cause-and-effect sequence in 
which certain program activities are the in- 
stigating causes and certain social [or envi- 
ronmental] benefits are the effects they 
eventually produce, (p. 102) 

Given this, impacts may be defined as longer-term 
effects or benefits that result from the activities in a par- 
ticular program (e.g.. long-term effects on program par- 
ticipants, tangible benefits to educational programs and 
providers). The measurement of impacts is sometimes 
referred to as impact assessment or impact evaluation. 
For example, with respect to ARE and EE programs, 
longer-term, cumulative effects may accrue from learn- 
ers' repeated exposure to natural environments through 
recreational activities such as canoeing and fishing 
(Sward and Marcinkowski 2001). Similarly, benefits of 
teacher in-service programs may extend from teacher- 
participants to the schools or programs in which they 
work (e.g., longer-term curricular change), and in some 
instances even to the natural environment (e.g., resource 
stewardsiiip). 

Program Logic and Logic Models 

The ability to tie a particular set of program activi- 
ties to learning outcomes and then to longer-term im- 
pacts in a convincing cause-and-effect manner requires a 
substantial amount of work to link theory, research, 
practice, and evaluation. Attempts by program evalua- 
tors to do so for a particular program lead to what they 
call program theory or program logic for that program. 



A critical aspect of program theory is how 
the various steps and functions [of a pro- 
gram] relate to each other. Sometimes those 
relationships involve only the temporal se- 
quencing of key program activities and their 
effects ... In other cases, these relationships 
have to do with activities or events that must 
be coordinated ... Other relationships entail 
logical or conceptual linkages, especially 
those represented in program impact the- 
ory.... Describing program theory, therefore, 
requires an understanding of how different 
events, persons, functions, and other ele- 
ments represented in the theory are pre- 
sumed to be related. (Rossi et al. 1999, pp. 
171-172). 

In evolving their ideas about program theory and 
program logic, some program evaluators have come to 
view programs from a systems theory perspective (i.e., 
as consisting of inputs, throughputs, and outputs). For 
any given program, they work to identify necessary in- 
puts (e.g., people, resources, internal and outside fac- 
tors), throughputs (e.g.. program activities), and their re- 
lationship to desired outputs (i.e., as described above, as 
outputs, outcomes, and impacts). As they identify these 
elements and relationships, they construct and make ex- 
plicit the logic within that program. They refer to the set 
of established, apparent, and hypothesized relationships 
among these elements as program logic. 

More specifically, Funnell (1997, as cited in Rossi 
et al. 1999. p. 1 72). McLaughlin and Jordan (1999), 
Rogers (2000), and others have described the elements 
and relationships that are central to program logic. Fun- 
nell (1997) identified the first of these as the cause-and- 
effect hierarchy of desired program outputs that lead to 
immediate outcomes, and in rurn to longer-term impacts. 
Other central features of program logic identified by 
these writers include: factors internal and external to a 
program and program activities that are likely to influ- 
ence these outcomesdmpacts, means for defining and 
measuring these outcomes/impacts, actual program per- 
formance data pertaining to these outcomes/impacts, and 
means for interpreting these performance data. 

The graphic depiction of these elements and the re- 
lationships for a given program is commonly referred to 
as a logic model. 

A rather common way of depicting the or- 
ganizational plan [or logic] of a program is 
in terms of inputs, representing the re- 
sources and constraints applicable to the 
program, and activities, indicating the ser- 
vices the program is expected to provide. 
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When included in a full logic model, these 
schemes typically represent receipt of ser- 
vices ... as program outputs, wliich, in turn, 
are related to desired outcomes. (Rossi et al. 

1999, p. HI). 

In addition to serving as communication devices, 
logic models may also serve as heuristic devices; that is, 
they allowing and even encourage a variety of questions 
inherent in program logic and program evaluation to be 
raised and addressed. Rossi et al. (1999) identity a num- 
ber of questions that may be posed as part of a review of 
the logic and plausibility of a program: 

• Are the program goals and objectives well defined? 

• Are the program goals and objectives feasible; i.e.. 
is it reasonable to assume that they can actually be 
attained as a result of program action? 

• Is the change process presumed in the program the- 
ory plausible? 

• Are the program procedures for identifying mem- 
bers of the target population, delivering service to 
them, and sustaining that service through comple- 
tion well defined and sufficient? 

• Are the constituent components, activities, and 
functions of the program w'ell defined and suffi- 
cient? 

• Are the resources allocated to the program and its 
various components and activities adequate? (pp. 
178-180) 

Use of Logic Models in This Paper 

l will use the term logic model in two related ways. 
First, I will refer to the graphic depiction of elements 
and relationships in Figure 1 as a general logic model. 
As a general model, it reflects common features of edu- 
cational programs in the U.S., but does not include all 
elements central to program logic and logic modeling 
(e.g., factors internal and external to a given program). 
The elements of this general logic model are: 

• Educational Purposes, which includes aims, goals 
and objectives; 

• Educational Tools, which includes curricular mate- 
rials; 

• Support Systems for Use of Those Tools: which in- 
cludes manuals, training, and technical support; 

• Delivery Systems, which includes sectors and target 
audiences; 



• Delivery Strategies, wliich includes the means and 
methods used to deliver education and information 
programs and activities; 

• Outputs, which are program-related number counts 
and feedback; 

• Outcomes, which are learning outcomes for pro- 
gram participants; 

• Benefits or Impacts, which are longer-term program 
payoffs. 

Second, I will refer to the graphic depictions of the 
EE program featured in this paper as program-specific 
logic models (Figure 2 and Figure 3). The use of general 
and program-specific logic models should help to make 
explicit the evolving logic inherent in this program, and 
allow it to be compared to other programs in the fields of 
ARE and EE. 



Introduction to the Program Featured in This Paper 

This paper will summarize a program directly re- 
lated to the stewardship emphasis of this document. This 
program is the issue and action instruction program de- 
veloped by Dr. Harold I-Iungerford and his colleagues af- 
filiated with Southern Illinois University at Carbondale 
(SIU-C) and, more recently, the Center for Instruction, 
Staff Development, and Evaluation (C1SDE). The initial 
curriculum they have been evolving since 1973 is now 
entitled Investigating and Evaluating Environmental Is- 
sues and Actions (1EEIA) (Mungerford, Litherland, Pey- 
ton, Ramsey. 8i Volk 1988, 1990, 1992, 1996). This and 
related curricula will be described in a section of this pa- 
per entitled "Issue and Action Instruction Curricular Ma- 
terials.” 



These curricular materials have been designed for 
use in formal school programs, although some of them 
have been adapted for use in non-formal programs (e.g., 
extended summer programs offered by an environmental 
center: see Jordan, I-Iungerford and Tomera, 1986). With 
respect to their target audience, the original issue and ac- 
tion materials w'ere designed for use at the middle/junior 
high level. As will be described later, this curriculum 
was permutated for use in upper elementary grades (i.e., 
as "extended case studies”), and most recently for use at 
the secondary level. Under "Delivery System” (Figure 
1), these are the audiences within the formal and noil- 
formal sector targeted by this program. 
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Figure 2. A Program-Specific Logic Model for 1EEIA and JESTS1S 
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Origins of This issue and Action Instruction Pro- 
gram 

This program and the curricular materials associated 
with it began in 1972 when Mr. Ralph Litherland, a 
middle grades science teacher, asked Dr. Harold Hun- 
gerford, a science education professor, for assistance. 
Mr. Litherland was attempting to involve students in 
multiple sections of his course in investigations of com- 
munity problems and issues. He found it difficult to fa- 
cilitate individual investigations for a large number of 
students without first equipping them with the thinking 
and inquiry skills to do so. They found that: 

... middle school students are ill-prepared to re- 
search environmental problems using primary or 
secondary sources of information (or both). 
Early attempts to provide a learning climate in 
which normal middle school students could ap- 
ply their own environmental interests to a re- 
search-oriented strategy failed miserably, even 
under the direction of highly competent instruc- 
tors. The students simply did not have the skills 
necessary for a successful interaction with envi- 
ronmental problems. (Hungerford and Litherland 
1973a, p.TM-1). 

Together they developed a set of skill-oriented, struc- 
tured modules "to provide those skills so that students of 
varying abilities can become truly autonomous learners" 
and to prepare "middle school students to investigate en- 
vironmental problems in an autonomous manner" (Hun- 
gerford & Litherland, 1973a, p. TM-1). The modules 
comprising this set were: 

Module I: Looking into Environmental Problems; 

Module II: Using Secondary Sources in Studying En- 

vironmental Problems; 

Module III: Using Surveys, Questionnaires, and Opin- 
ionnaires in Environmental Science; 

Module IV: Interpreting Data in Environmental Sci- 
ence. 

According to Hungerford and Litherland (1973a, p. TM- 
1) "They set the scene for research and make provisions 
for research-no more." 

The first four modules are higlily structured 
- purposefully structured in an attempt to 
produce a successful learning hierarchy. The 
fifth module, however, changes the rules 
completely ... Module V is both a self-paced 
and individualized learning experience 
(Hungerford and Litherland 1973a, Preface). 



In short, “Module V: Studying an Environmental Prob- 
lem: It's Your Move/' was designed to support students 
in the type of autonomous research or investigation for 
which Modules I - IV had been preparing them. Thus, 
from the earliest days of this program, its developers saw 
the need for structured skill development as a prerequi- 
site to autonomous skill application. Further, Hungerford 
and Litherland also ... 

... hoped that this research [autonomous stu- 
dent investigations of local environmental 
problems] will set the scene for environmental 
action on the part of students. If it does, 
the modules will have been eminently suc- 
cessful - beyond the primary intent of the au- 
thors (1973a, p. TM-1). 

However, there was little, if any, provision for this 
in the 1973 and 1975 editions of this curriculum. Clear 
and careful attention to this did not appear until the 1978 
edition. From the outset, the developers also saw the 
need for program evaluation, and began using evaluation 
as a tool to both improve this program and articulate its 
strengths (i.e., validate it). 

During the 1972-73 school year. Modules I, 

II, III, and V were used with 130 students at 
Lincoln Junior High in Carbondale, Illinois. 

Results of that testing have prompted minor 
revisions of the original materials. Of even 
greater significance is the observation that 
interaction with Modules L II, and III did. in 
fact, produce a cognitive climate favorable 
to autonomous research ... many of the stu- 
dents engaged in training and research did, 
in fact, evaluate their own values toward 
numerous issues of environmental sig- 
nificance with subsequent revision of those 
values taking place (Hungerford and Lither- 
land 1973a, p. TM-2). 

In addition to well-designed materials and program 
evaluation, the developers recognized that a third factor 
would be critical to the success of this program: well- 
prepared teachers. "Like so many teaching strategies, the 
success of this modular design depends to a large extent 
on the teacher" (Hungerford & Litherland, 1973a, p. 
TM-2). It became apparent to the developers that the 
shift from the structure of Modules I - IV to the indi- 
vidualized pace of Module V could be problematic for 
teachers: “The teacher MUST be prepared for this or 
his/her psyche will be ... traumatized by the shock of go- 
ing from a highly structured situation into a highly un- 
structured one" (Hungerford & Litherland, 1973a, Pref- 
ace). They also realized that some teachers would lack 
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the willingness and/or ability to shift from one teaching 
style and classroom environment to the other. Wiiile it 
would be some years before the developers and their col- 
leagues would evolve a teacher education program and 
network, the need for it was apparent from the outset. 

Overview of The Issue and Action Instruction Pro- 
gram 

The primary focus of this program and associated 
curricula is on the development and application of inves- 
tigation and evaluation skills to environmental problems 
and issues, and to alternative solutions and action strate- 
gies. As has been described elsewhere (Bardwell. Mon- 
roe and Tudor 1994; Ramsey 1998), these materials take 
a structured approach to the development of these skills, 
and then encourage the teacher to become a guide and 
facilitator during skill application. Traditionally, teach- 
ers who work with tills program approach skill applica- 
tion in the form a project (i.e., for individuals, coopera- 
tive groups, or whole classes), and refer to this project as 
an "issue investigation." These projects require students 
to sequence and apply the skills they have been develop- 
ing as they investigate community-based problems and 
issues. Ideally, these investigation projects lead into and 
culminate in action plans and projects that target the 
problems and issues investigated by students. For the 
most part, this program has been unique in maintaining 
its primary focus on skill learning, while allowing and 
encouraging students to engage in content learning on a 
"just-in-time" basis (i.e., substantial content learning oc- 
curs during the aforementioned projects). More will be 
said about the importance of skill learning in the section 
of this paper on the "Purposes of This Issue and Action 
instruction Program." 

As with many outdoor, conservation and environ- 
mental education programs (e.g., Hooked on Fishing; 
Project Learning Tree; Project WILD; Project WET), 
this program also has evolved a significant teacher 
preparation component and network of certified trainers. 
However, there are several important differences be- 
tween the teacher preparation component for this pro- 
gram and those commonly used elsewhere (e.g., six hour 
or one day workshops). These differences pertain to the 
purpose(s), nature, length, and costs of the workshops, as 
related to (or determined by) the nature of the program 
and curricula. These differences also will be described in 
greater detail in the section of this paper entitled 
"Teacher Preparation for Issue Instruction." 

There is an additional way in which this issue and 
action instruction program is different than most other 
outdoor, conservation, and environmental education 
programs (see Disinger 1981; Chenery and Hammerman 
1984; Rakow and Lehtonen 1988; Simmons 1991). By 



the late 1980s and early 1990s, Hungerford and his col- 
leagues had evolved a substantial research and class- 
room evaluation base to support claims about this pro- 
gram’s effectiveness and worth with respect to steward- 
ship behavior. The former will be summarized in the 
section of the paper entitled "Research Base for This 
Program." while the latter will be summarized in the sec- 
tion of the paper entitled "Classroom Evaluation Base 
for This Program." The evolving program theory that 
links these two elements is reflected in Figures 2 and 3. 

Purposes of This Issue and Action Instruction Pro- 
gram 

While all issue and action instruction curricula as- 
sociated with this program feature the same skill devel- 
opment and application sequence, the educational pur- 
poses (i.e., mission, aims, goals, and objectives) they 
serve are broader than that. As suggested by the logic 
model in Figure 1, to understand this program and these 
curricula, it is necessary to understand these “Purposes." 
The broadest statement of purpose will be referred to as 
a mission statement. Typically, a program’s mission 
statement is a broad philosophical statement about what 
it hopes to contribute to without any tying that to any 
particular age or grade level (i.e., as in goals), or any set 
of activities (i.e., as in objectives). Hungerford, Peyton, 
and Wilke (1980b) adapted the following super-ordinate 
goal statement from Harvey's dissertation (1976) to 
serve as their mission statement: 

The Superordinate goal ... to aid citizens in 
becoming environmentally knowledgeable 
and. above all, skilled and dedicated citizens 
who are willing to work, individually and 
collectively, toward achieving and/or main- 
taining a dynamic equilibrium between 
quality of life and quality of the environ- 
ment (p. 43). 

This statement appears in numerous articles prepared by 
the developers to describe the purposes of this program 
(Hungerford and Volk 1984, 1990; Ramsey, Hungerford 
and Volk 1992). It clearly reflects the role of citizens in 
resource stewardship and, as such, provides guidance for 
what the aims, goals, and objectives to be derived from it 
should address. 

The aforementioned articles also articulate the edu- 
cational goals of this issue and action instruction pro- 
gram. These four goals were drawn directly from Goals 
for Curriculum Development in Environmental Educa- 
tion (Hungerford, Peyton, and Wilke 1980b) that, in 
turn, were validated against the Tbilisi objectives 
(Unesco 1978). Tiiese four goals (or goal levels) are: 
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Level I. Ecological Foundations: This level 
seeks to provide the receiver [learner] with 
sufficient ecological foundations knowledge 
to permit hiirt/her to eventually make ecol- 
ogically sound decisions with respect to en- 
vironmental issues. 

Level II. Conceptual Awareness Level - Is- 
sues and Values: Tills level seeks to guide 
the development of a conceptual aw'areness 
of how individual and collective actions 
may influence the relationship between 
quality of life and the quality of the envi- 
ronment and, also, how these actions result 
in environmental issues which must be re- 
solved through investigation, evaluation, 
values clarification, decision making, and 
finally, citizen action. 

Level 111. Investigation and Evaluation 
Level: This level provides for the develop- 
ment of the knowledge and skills necessary 
to permit receivers [learners] to investigate 
environmental issues and evaluate alterna- 
tive solutions for remediating those issues. 
Similarly, values are clarified with respect 
to those issues and alternative solutions. 

Level IV. Environmental Action Skill Level 
- Training and Application: This level seeks 
to guide the development of those skills 
necessary' for receivers to take positive envi- 
ronmental action for the purpose of achiev- 
ing and/or maintaining a dynamic equilib- 
rium between quality of life and the quality 
of the environment (Hungerford, Peyton, 
and Wilke 1980b, pp. 43-44). 

Of these four goal levels, Levels II, III and IV re- 
ceive substantial attention in all issue and action instruc- 
tion curricula. In Investigating and Evaluating Environ- 
mental Issues and Actions (IEEIA), Level 1 is not ad- 
dressed due to the impractical ity of providing scientific 
background on the wide range of environmental prob- 
lems and issues it introduces. Rather, as was done by 
Ramsey during his years as a middle level science 
teacher, teachers using IEEIA are encouraged to teach a 
unit on ecology beforeliand. In some other issue and ac- 
tion instruction curricula. Level I has been expanded be- 
yond "Ecological Foundations” to include the other rele- 
vant "Scientific Foundations" that pertain the theme of 
that curriculum. This will be further clarified in the next 
section of this paper. 

As is common to many, but not all, educational pro- 
grams and curricular materials, all of the issue and action 



instruction curricula associated with this program con- 
tain instructional objectives. Typically, developers have 
prepared instructional objectives for each chapter and 
are found on the first or second page of each chapter in 
these curricular materials. These objectives were devel- 
oped and presented to support instructional planning and 
delivery, as well as assessment (i.e., on a pre- 
assessment, formative, and/or post-assessment basis). 
Readers interested in learning more about these instruc- 
tional objectives are encouraged to review these curric- 
ula. The above mission statement, and their associated 
goals and objectives have guided the development, im- 
plementation, and evaluation of all curricular materials 
associated with this issue and action instruction program. 

The only aspect of "Purpose" not addressed above is 
the educational aims of this program. Aims can be 
viewed as characteristics to be developed or acquired 
over a lifetime (i.e., beyond the years of K-12 school- 
ing). In the EE literature, when reference is made to 
characteristics of an environmentally literate citizen or 
citizenry, those characteristics can and should be viewed 
as aims of EE. The goals and objectives of EE programs 
such as this one should help learners develop or acquire 
these characteristics (i.e., meet these aims). The mission 
statement (superordinate goal) noted above introduces 
several of these aims: knowledgeable, skilled, dedicated, 
and willing to work. These aims are commonly found in 
environmental literacy frameworks constructed by the 
developers (Hungerford and Tomera 1985; Hungerford 
and Volk 1990) and by others in the field of ££ (Sim- 
mons 1995; Wilke 1995). 

Modern models of environmental literacy have 
evolved over much of the same time period, as did 
IEEIA. In part, this was due to the fact that some of the 
research and evaluation studies associated with this issue 
and action instruction program helped to shape those 
models of environmental literacy (e.g., Ramsey 1979, 
1989, 1993; ftlingler 1980; Hines 1985; Sia 1985; Sivek 
1989). "Nonetheless, w'hile the components of this evolv- 
ing model of environmental literacy (aims) w'ere not 
available to guide the early development of IEEIA. they 
have guided assessments done as part of classroom 
evaluation studies, and guided both later revisions of 
IEEIA and the development of more recent issue and ac- 
tion instruction curricula (see Hungerford. Litherland, 
Peyton, Ramsey, and Volk 1988, 1990. 1992, 1996; 
Ramsey, Hungerford, and Volk 1989; Marcinkowski et 
al. 2000). From the perspective of the general logic 
model (Figure 1), this is precisely how the developers 
have built their program-specific logic model: by refin- 
ing program "Purposes" and "Tools” so that they would 
be more clearly reflected in program "Outcomes" and 
"Impacts.” Given this, it is appropriate to list here the 
characteristics of an environmentally literate citizen 
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viewed as aims of this program (Hungerford, Litheriand, 
Peyton, Ramsey, and Volk 1996, p. 205): 

• Environmental sensitivity (i.e., feelings of 
comfort in and empathy toward natural ar- 
eas); 

• Knowledge of ecological concepts; 

• Knowledge of problems and issues; 

• Skill in identifying, analyzing, investigating, 
and evaluating problems and solutions; 

• Beliefs and values (i.e., beliefs are what in- 
dividuals hold to be true, and values are 
what they hold to be important regarding 
problems/issues and alternative solu- 
tions/action strategies); 

• Knowledge of action strategies; 

• Skill in using action strategies; and 

• Internal Locus of Control (i.e., the belief 
and/or feeling that working alone or with 
others, an individual can influence or bring 
about desired outcomes through her/his ac- 
tions). 

Issue and Action Curricula 

According to the general logic model presented in 
Figure 1, the issue and action curricula to be described 
below all serve as part of the "Tool Box.” What are these 
tools, and how have they evolved since the 1973 trial 
edition? 

Investigating and Evaluating Environmental issues and 
Actions: Skill Development Modules 

The previously described trial education was the 
first in a series of editions of the modules designed to 
promote development and application of environmental 
problem-solving skills. In 1973, the developers prepared 
a revised edition based on the results of the field test 
(Hungerford & Litheriand 1973b), These modules were 
revised again two years later, in this case with the sub- 
stantial addition of artwork by then current graduate stu- 
dents (Hungerford and Litheriand 1975), 

It was not until the later 1970s that these modules 
saw the first of several expansions. With the assistance 
of Dr. Audrey Tomera, and insights gleaned from the 
dissertation work of R. Ben Peyton ( 1 972) in the area of 
environmental action, the developers expanded these 
materials to include a new module, ’’Module VI: Envi- 
ronmental Action Strategies" (Hungerford, Litheriand. 
Peyton, and Tomera 1978), In line with previously men- 
tioned action-oriented aims and goals, the purpose of 
this module was to provide students with an opportunity 
to develop conceptual knowledge of and skill with re- 



sponsible environmental action strategies. Consequently, 
Module VI addressed four major components of their 
"Paradigm for Citizen Responsibility": (a) whether spe- 
cific human actions were positive, negative, or passive 
with respect the natural environment; (b) five categories 
of environmental action; (c) different levels at which ac- 
tions could be taken: and (e) tliirteen questions, referred 
to as action analysis criteria, for evaluating the appropri- 
ateness of actions proposed or planned by students 
(Hungerford and Peyton 1980). At the end of Module 
VI, students were required to prepare and evaluate plans 
for an action based upon the findings and recommenda- 
tions from their investigation, but were not required to 
carry out that action: 

Your assignment here is NOT to take action, 
because this is something YOU would have 
to do AS A CITIZEN. Whether you take 
action will depend on your values and 
whether your work in these modules has 
persuaded that you should become involved 
... The final decision is yours to make (Hun- 
gerford etal. 1978, p. 166). 

The addition of Module VI went a long way toward real- 
izing the hopes offered by Hungerford and Litheriand 
(1973b) noted earlier. 

The next edition of the Modules was completed in 
1985. By then, the field of EE had experienced a surge 
of growth and development, all of which influenced this 
program. Developments during this period included pub- 
lication of the reports of major conference sponsored by 
Unesco (Unesco 1977, 1978), completion of disserta- 
tions that helped define the field (Harvey 1976; Hart 
1980). curriculum goals for EE that drew' from and ex- 
panded on these sources (Hungerford, Peyton and Wilke 
1980b), the first publications of the Research Commis- 
sion that North American Association for Environmental 
Education (NAAEE) established in 1980 (lozzi 1981, 

1984) , early models of environmental literacy (Hunger- 
ford and Peyton 1976; Hungerford and Tomera 1977, 

1985) and, finally, seminal dissertation studies on corre- 
lates and predictors of responsible environmental behav- 
ior (Hines 1985, Sia 1985). 

During this same period, Hungerford and Tomera 
served as major advisors for a series of Masters theses 
and research projects that served as the first experimen- 
tal-type evaluation studies of this issue and action in- 
struction program (Ramsey 1979, Klingler 1980, Volk 
1980, Childers 1981). Of these authors, Ramsey and 
Volk went on to make significant contributions to the 
1985 edition of this curriculum based on their work in 
these studies, their classroom experiences with this cur- 
riculum, and their involvement in developing other cur- 
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riculum components (e.g., issue analysis procedures; see 
Volk 1980, Ramsey, Hungerford and Volk 1989). 

With the completion of the 1985 edition, now enti- 
tled Investigating and Evaluating Environmental Issues 
and Actions (IEE1A; Hungerford, Litherland, Peyton, 
Ramsey, Tomera and Volk 1985), this issue and action 
instruction program was coming of age. This curriculum 
reflected key aims and goals for the field of EE, and was 
supported by both positive findings from classroom 
evaluation studies and a growing body of research per- 
taining to environmental literacy and behavior. With a 
sound curriculum and program, questions were raised 
about how the skill development and application se- 
quence that served as the core of this curriculum and 
program could be made available to a wider range of 
students. Two responses to this question were offered in 
the later 1980s. 

Permutations of IEEIA 

The first response reflected the developers* in- 
volvement in the science education community. In par- 
ticular, Ramsey pushed for a Science-Technology- 
Society (STS) application (Hungerford, Volk and Ram- 
sey 1990). The developers did this to address the growth 
of STS in the 1980s within the science education com- 
munity, and interest within the STS community in envi- 
ronmental problems and in skill development and appli- 
cation with respect to STS problems (Rubba and Weis- 
enmayer 1988, Coleman 1989, Simpson, McLaughlin, 
Volk and Hungerford 1989, Ramsey 1993). This STS 
format generally resembles IEEIA, with the following 
modifications: 

• A new' Chapter 1 was added to provide students 
with sufficient background on the nature of science, 
teclinology, and society, and on interactions 
amongst them; 

• The range of problems used in skill development ac- 
tivities was expanded beyond environmental ones to 
include a wider range of STS problems (e.g., health 
and biomedical problems, technology-oriented prob- 
lems such as agriculture and transportation); 

• In Chapter 7, the main focus of the chapter w'as ex- 
panded to the solution of STS problems and resolu- 
tion of STS issues. 

As mentioned earlier, a second response w'as under- 
taken to allow' the skills in LEELA to be developed and 
applied by students in lower grades, particularly grades 
5-6. In this case, a number of the developers and their 
colleagues helped prepare w'hat they referred to as the 
"Extended Case Study" format (e.g.. Culen, Simpson, 
Hungerford, Ramsey and Volk 1988. Ramsey, Hunger- 
ford and Volk 1989). This intermediate-level extended 



case study format has several important features, includ- 
ing: 

• It consists of four or five chapters, each of which 
addresses one goal level from Goals for Curriculum 
Development in Environmental Education (Hunger- 
ford, Peyton and Wilke 1980b); 

• The range of science and environmental content is 
narrow'ed to focus on a particular theme of national 
significance such as solid waste (Ramsey, Hunger- 
ford and Volk 1989, 1996), wetlands (Culen 1992), 
or threatened and endangered animals (Hagengruber 
and Hungerford 1993; Hungerford, Hagengruber, 
and Bluhm 1999); 

• The skill development sequence is somewliat lim- 
ited in comparison to IEEIA, allowing more space 
for the treatment of science and environmental con- 
tent pertinent to the curriculum's theme; and 

• Skill application is commonly undertaken in the 
context of the curriculum's particular theme, and in 
the form of whole class or large group investiga- 
tions (Ramsey and Hungerford 1989). 

As w'as true with IEEIA, several classroom-based 
evaluation studies of the extended case study format 
were carried out by developers for their dissertation 
studies (Simpson 1991, Culen 1996). And, as might be 
expected in light of positive results, extended case study 
materials have been both distributed and used in teacher 
in-service workshops since 1990 so as to provide trained 
teachers with a wader range of curriculum infusion op- 
tions. 

Lastly, a new' experimental version of the extended 
case study was developed to address new' opportunities 
and pressures in the public school that arose in the 1990s 
(Marcinkow'ski, Anderson, Drag, English, Lunsford, & 
Sward, 2000). This version of the extended case srudy 
format is designed for higher rather than lower grade 
levels (i.e., secondary level and higher). Altered features 
of this secondary-level extended case study format in- 
clude: 

• The use of a particular geographic region as its focus 
and theme (i.e., the Greater Everglades Watershed); 

• This theme allows the chapter on science/ecological 
foundations to focus on the natural history and ecol- 
ogy of this region; 

• This theme allows for a wider range of region- 
specific problems to be addressed than is typically 
found in intermediate-level extended case studies; 

• While keeping the more detailed skill development 
sequence of IEEIA intact, more theme-related sub- 
ject matter has been added to provide students with 
a stronger content background (i.e., in light of the 
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complexity of problems and restoration efforts 
within this watershed) and to provide greater cover- 
age of national and state standards that now drive 
school curricula; 

• Because both subject matter and skills vary from 
chapter-to-chapter, a sample issue investigation was 
added as a model. 

Much work remains to determine if this secondary-level 
extended case study format will be as effective as its 
predecessors. 

Teacher Preparation for Issue and Action Instruction 

The preparation of teachers to effectively use cur- 
ricula is a central concern of virtually all EE programs. 
Materials have been prepared to support these efforts 
(Wilke, Peyton, and Hungerford 1987; Hungerford, 
Volk. Dixon, Marcinkowski and Sia 1988; Marcin- 
kowski, Volk and Hungerford 1990; Simmons 2000). 
Due to its importance, a separate component that in- 
cludes teacher preparation and other forms of teacher 
support, "Support for Tool Use," appears in the general 
logic model (Figure 1). One may apply the general logic 
model to teacher preparation programs just as I am ap- 
plying it here to a K-12 program. In this case, the "Pur- 
pose" and "Outcome" of teacher preparation would be 
their understanding of and ability to effectively use issue 
and action curricula, and the "Tools" are those curricula. 
The target audience consists of practicing teachers in 
grades 6-12 and higher, and other interested education 
personnel. There are several "Delivery Systems" and 
"Delivery Strategies" for tliis type of teacher preparation, 
as will be described below. 

Crucial Delivery Strategies for Teacher Preparation 

As with most ARE and EE programs, teacher prepa- 
ration programs for issue and action instruction have 
evolved over time. Due to the nature of the skill devel- 
opment and application sequence inherent in ail issue 
and action curricula, the preparation of teachers to effec- 
tively use these curricula is substantially different than 
for many other nationally-available ARE and EE pro- 
grams. The following have become common features of 
virtually all efforts to prepare teachers to effectively use 
the issue and action curricula described earlier (see Fig- 
ure 2 and Figure 3). 

First, teacher preparation programs must be longer 
in duration. Tliis is, at least in part, due to the fact that 
few teachers have learned these skills, much less how to 
apply them or even teach them, during their pre-service 
training (Peyton 1978, Peyton and Hungerford 1980, 
McKeown-Ice 1995). For many years, the average 
length of teacher in-service workshops in this program 



was two weeks (i.e., 10-12 days). When funding was 
made available from larger grant sources (e.g., NSF, 
U.S. Department of Education's National Diffusion 
Network), summer workshops lasted as long as 3-4 
weeks. Over the last few years, in response to concerns 
about length, Hungerford. program developers, and other 
certified trainers have designed and run workshops last- 
ing one week. 

Second, because the focus of this program was on 
skill development and application, and the general use of 
these materials requires a rather lengthy instructional se- 
quence, the nature of this teacher preparation was differ- 
ent (i.e., it could not and did not rely on the use of a set 
of largely unrelated activities). From the outset Hunger- 
ford and his colleagues took the position that all teacher 
preparation programs should model both the skill devel- 
opment and the skill application process. The former 
helps to familiarize teachers with the skills, the curricu- 
la r material, and teaching methods appropriate for the 
teaching/leaming of skills. The latter requires that teach- 
ers, usually in small groups, become familiar with skill 
application, the project sequence, and methods for facili- 
tating these projects. By the end of each teacher work- 
shop, all teacher-participants are familiar with these 
skills and have reported out their own issue investigation 
project. 

Third, due to difficulties teachers often encounter in 
infusing or inserting an issue and action curriculum into 
her/his school program, in implementing (teaching from) 
that curriculum, and in appropriately guiding projects 
and assessing student learning, support for these teachers 
is almost always needed after in-services have been 
completed. This can take several forms: (a) encourage- 
ment to teachers to use Teachers Guides that accompany 
all issue and action curricula; (b) technical support 
through the workshop providers and/or material devel- 
opers (e.g.. CISDE); and (c) follow-up visits to teachers 
in their classrooms. Because no one of these was suffi- 
cient unto itself, the developers have relied upon a com- 
bination of all of these, particularly when time and fund- 
ing permitted. 

Fourth, due to the duration of these workshops and 
the need for follow-up, they tend to be more costly to 
run. Usually two or even three certified trainers oversee 
each workshop, raising salary and travel support costs. 
Teacher-participants are usually provided with housing 
and meals for the duration of the workshop, as well as 
travel support to get to and from the workshop site. Fol- 
low-up visits often require additional funding for staff 
time and travel costs. Nonetheless, Hungerford and his 
colleagues would argue that cost should not be the pri- 
mary determinant of workshop length or follow-up (i.e., 
as may be argued from an output or cost/benefit perspec- 
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five), because the benefits to teachers and. through them, 
to their students, exceed those possible in shorter work- 
shops and without sucli follow-up. Given tliis, grant pro- 
grams have become likely sources for this kind and level 
of support. 

Viable Delivery Systems for Teacher Preparation 

Because the above features are critical conditions to 
prepare teachers to effectively use issue and action cur- 
ricula, there were three viable options to prepare teach- 
ers to effectively use them. First, in the 1970s, Hunger- 
ford and Tomera taught this skill development and ap- 
plication sequence in the form ot a graduate-level EE 
course. Virtually all of the individuals cited in this paper 
as additional curriculum developers, authors of research 
studies, and authors of classroom-evaluation studies re- 
ceived their initial training in this manner. Many of these 
former Southern Illinois University graduate students 
have gone on to teach in other universities and to offer 
graduate-level issue and action instruction courses of 
this kind. 

A second option was to offer issue and action in- 
struction training in the form of an undergraduate course 
for pre-service teachers. It is noteworthy that such an is- 
sue and action methods course has been offered as a re- 
quired science methods course at Southern Illinois Uni- 
versity since 1989. Unfortunately, this appears to be the 
only university in the U.S. in which an issue and action 
course is included in a pre-service teacher education 
program as either a required or an elective course 
(McKeown-Ice 1995: T. Volk, personal communication. 
March 9, 2001). 

Due to above-mentioned limitations in pre-service 
teacher education programs, the third option for prepar- 
ing teachers is the most widely used in the field of EE: 
teacher in-service workshops. Since the early 1980s, the 
developers have obtained funding through State Eisen- 
hower Funds and other sources to run in-service regional 
workshops in Illinois and other midwestem states. Once 
the 1985 edition of IEEIA was completed, efforts to ob- 
tain grants through federal agencies for these in-service 
trainings were particularly successful. Between 1985 and 
1994, the developers obtained no less than seven grants 
through the National Science Foundation, allowing in- 
service workshops to be provided for teachers from 
Midwestern, Mid-Atlantic, and Southern states. These 
grants supported lengthy trainings and periodic follow- 
up efforts, and required substantial assessment and 
evaluation efforts. Data from these grant programs, cou- 
pled with data from classroom evaluation studies, were 
sufficient to permit IEEIA to be adopted onto the U.S. 
Department of Education's National Diffusion Network 
(NDN) in 1990, and the STS version to be adopted a 



year later. Through NDN, the developers garnered sev- 
eral years of support for material dissemination and 
teacher in-service workshops. Lastly, as a result of the 
growing credibility of this issue and action instruction 
program, several of the developers were approached by 
Kraft Foods to prepare a solid waste curriculum (Hun- 
gerford, Volk and Ramsey 1992, 1993) and to provide 
in-service training to selected teachers. 

Collectively, the in-service workshops funded by 
private sources, and state and federal agencies aft owed 
the developers to build a national network ot 33 nation- 
ally certified IEEIA trainers and a larger cadre of dis- 
trict-level trainers in at least 13 states. On a national 
scale, trainers in this network are responsible for almost 
all of the program-related teacher preparation associated 
with the above three options. 

Research Base for This Issue and Action Program 

The research base for a particular program is impor- 
tant with respect to the general logic model even though 
there is no separate element for it (Figure 1). This is be- 
cause research provides empirical evidence about the re- 
lationships between two or more ot the model's ele- 
ments. As noted earlier, the relationship of "Impacts" to 
other features of a logic model is ot central importance, 
and therefore will be given primary attention in this sec- 
tion. As professionals associated with a particular pro- 
gram accumulate research evidence regarding the rela- 
tionships between "Impacts" and other model elements, 
they may use that evidence to evolve a program theory 
for that program. 

The theory and research base for any education pro- 
gram may draw upon research carried out by profession- 
als related to that program, as well as upon the literature 
in related fields. This has been true for this issue and ac- 
tion instruction program. Its developers have relied upon 
their own studies (Hungerford, 1988: Hungerford, Volk 
and Ramsey 2000), research in EE (lozzi 1981, 1984), 
and research in other fields (Hines 1985) as they evolved 
and refined their program. It is highly significant that, 
over the past two decades, the developers and their col- 
leagues have come to recognize that the "program the- 
ory" that underpins and drives a good bit of ARE and EE 
practice is not supported by this research. 

The Knowledge-Attitude-Behavior Model 

The most popular program theory operating in EE 
and, quite possibly ARE, is known as the knowledge- 
attitude -behavior (KAB) model. The kinds of behavior 
(B) of interest here are the wide range of stewardsliip 
behaviors that are collectively known as Responsible 
Environmental Behavior (REB). Unfortunately, the 




110 



Tom Marcinko wski 



An Issue and Action Instruction Program 



growing body of research about REB both within and 
outside the field of EE clearly demonstrates that this is 
an overly simplistic model: the K-A-B relationships are 
neither direct nor linear, and there are more variables 
and phases involved in the process of forming or shaping 
REB. As a result, this model has been critiqued by Pey- 
ton (1981), Sia (1985), Hines (1985), Marcinkowski 
(1989), Hungerford and Volk (1990), and Simmons 
(1991). 

What kinds of theoretical assumptions are reflected 
in the KAB model? Hungerford and Volk (1990, p. 9) 
quote Ramsey and Rickson (1976, p. 10) as indicating 
that "increased knowledge leads to favorable attitudes ... 
winch in turn lead to action promoting better environ- 
mental quality" (emphases added). Marcinkowski (1989, 
p. 95) offered a similar quote from Birch and Schwaab 
(1983, p. 30) to further illustrate the logic of this model: 

As previous research indicates ... the as- 
sumption must be made that informed atti- 
tudes will lead to subsequent water conser- 
vation behavior ... Further research should 
attempt to offer empirical evidence that 
knowledge and attitude gains resulting from 
the water conservation unit wall influence a 
student's behavior regarding efficient water 
use" (emphases added). 

Finally, in analyzing the goals of 1,225 nature and 
environmental centers, Simmons (1991) reported that 
about two-thirds of the responding centers identified 
goals which reflected knowledge-behavior and/or K-A-B 
models. These anecdotes provide some insight into the 
nature and pervasiveness of K-A-B thinking in the field. 

lozzi (1981, 1984), Hines (1985). and others have 
reported that attitude is by far the most commonly stud- 
ied variable in EE, a circumstance which both reflect the 
pervasiveness of thinking about the role of attitudes (i.e., 
within the K-A-B model). They have also provided the 
research evidence to assess and critique this role. With 
respect to the knowledge-attitude (K-A) portion of the 
K-A-B model, Peyton (1981) presented a review of 21 
studies that investigated these relationships. His review, 
and Marcinkowski’s subsequent research review (1989, 
pp. 94-97), revealed a more complex relationship be- 
tween knowledge and attitude than is implied in the K- 
A-B model. Research evidence indicates that: (a) the 
strength of the K-A relationship is contingent upon the 
particular dimension(s) of knowledge and attitude of in- 
terest; and (b) even when these are well matched, tlxis re- 
lationship may be non-linear and can change over time. 
Similarly, review's of research on the attitude-behavior 
(A-B) relationship (Sia 1985, pp. 66-67; Marcinkowski 
1989, pp. 97-101) indicated that: (a) as with K-A, the 



strength of the A-B relationship is contingent upon the 
dimension(s) of both attitude and behavior; (b) the 
strength of this relationship also appears to be contingent 
upon or an artifact of measurement conditions; and (c) 
while attitude is one of several variables influencing be- 
havior, there appear to be other variables vvliich may 
moderate or influence the attitude-behavior relationsliip. 

Evidence to support critiques of the K-A-B model 
can be found in other bodies of research, including 
Fishbein and Ajzen's work (1975, 1977) and research in 
the "diffusion-adoption" tradition (Rogers, 1995). 
Rogers reported that "KAP studies are sample surveys of 
knowledge (K), attitudes (A), and adoption of the prac- 
tice (P) of family planning innovations ... [Mjost na- 
tional family planning programs have found it much eas- 
ier to diffuse knowledge about contraceptive methods 
(K) and to achieve favorable attitudes toward family 
planning (A), than to secure the widespread adoption 
and practice of contraception (P) by the target audience. 
Thus, KAP surveys often find a ’KAP-gap' with a rela- 
tively high percentage of knowledge and favorable atti- 
tudes toward family planning methods ... but a relatively 
low rate of adoption ... Knowledge gain and attitude 
change can be achieved more easily than can adoption of 
family planning methods” (1995, pp. 70-71; emphases 
added). Review's of the research literature within and 
outside the field have clearly indicated the K-A-B model 
was a faulty and inadequate program theory to guide EE 
practice. If REB is to given more than lip service, a more 
adequate knowledge base had to be constructed through 
research to inform and guide practice. 

Evolving Research Base for This Issue and Action 
Program 

Over the past two decades, the developers of this 
program have made a concerted effort to summarize and 
synthesize research that has a bearing on this program 
(Hines and Hungerford 1984, Hines 1985, Hungerford 
1988, Hungerford and Volk 1990. Volk and McBeth 
1997, Marcinkowski 1998, Hungerford. Volk and Ram- 
sey 2000). 

Of these research documents, three made notewor- 
thy contributions to the research base and program the- 
ory underlying this program. The first of these is the dis- 
sertation study by Hines (Hines 1985; Hines. Hungerford 
and Tomera 1986/87). While this w'as not the first at- 
tempt to summarize or synthesize the research on vari- 
ables related to REB (sec Lipsey 1977; Cook and Ber- 
renberg 1981; Hines and Hungerford 1984), it is the 
most recent and remains the only quantitative synthesis 
or meta-analysis of research in the area of REB and in 
tiie field of EE. Following traditional meta-analysis pro- 
cedures, she reviewed studies on REB from a variety of 
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fields and used statistical procedures to determine the 
strength of relationship between REB and associated 
variables. The core of her findings is presented in Table 
1 . 

These findings provided evidence that knowledge 
and attitude were not as strongly correlated to REB as 
implied in the KAB model, and identified at least three 
other variables missing from that model: verbal com- 
mitment (or willingness, intention), locus ot control, and 
personal responsibility. 



Table 1. Summary of Hines’ Meta- Analysis Results 
for Descriptive Studies 



Corrected 
correlation 
Variable coefficient 


Corrected 

standard 

deviation 


Number 
of values 
based on 


Verbal commitment 


.491 


.130 


6 


Locus of control 


.365 


.121 


4 


Attitude 


.347 


.224 


51 


Personal responsibility 


.328 


.121 


6 


Knowledge 


.299 


.195 


17 


Education level 


.185 


.122 


11 


Income 


.162 


.084 


10 


Economic Orientation 


.160 


.1 18 


6 


Age 


-.151 


.200 


10 


Gender 


.075 


.084 


4 



Note: Several studies reported data on more than one 
variable. 



During the same time that Hines was carrying out 
her study, a fellow graduate student was carrying out a 
different kind of investigation of variables related to 
REB. Sia (1985; Sia, Hungerford and Tomera, 1985/86) 
had begun to explore predictors of REB, drawing on 
Hines 1 work, the results of prior prediction studies and a 
working model of environmental literacy prepared by 
the developers (Hungerford and Tomera 1977; 1985). 
Several things about his study and results are notewor- 
thy. First, he was one of the first to investigate more than 
one or two predictor variables; i.e., he investigated eight. 
Second, of these eight, six were significantly correlated 
to REB (r = -.39 to .56; p<.05): (1) environmental sensi- 
tivity; (2 and 3) knowledge of and skill in using envi- 
ronmental action strategies; (4 and 5) individual and 
group locus of control; and (6) psychological gender role 
(Sia et al. 1985/86, p. 37). Two of these variables lie at 
the heart of this issue and action program (i.e.. 2 and 3), 
while two others were identified by Hines and are also 



positively influenced by this program (i.e., 4 and 5). 
Third, the results of his final stepw'ise regression analy- 
sis (Table 2) indicated that skill in using action strate- 
gies, environmental sensitivity, and knowledge of action 
strategies accounted for the majority of the variance (or 
variability) in REB scores (i.e., 49.24% of 51.25%). 



Table 2 - Results of Sia’s Stepwise Regression Analy- 
ses for the Total Sample (n=171) 



Predictor R-Square 

Variables Values 


% Explained 
Variance 


F 


P 


Perceived skill in 
using env. action 
(df *= 1 . 169) 


.3454 


34.54% 


89.16 


.0001 


Level of env. 
sensitivity 
(df- 2, 168) 


.1292 


47.46% 


41.31 


.0001 


Perceived knowl. 
of env. action 
(df - 3, 167) 


.0178 


49.24% 


5.87 


.0165 


Belief in/attitude 
toward technology 
(df - 4, 166) 


.0097 


50.21% 


3.24 


.0737 


Psychological sex 
role classification 
(df - 7, 163) 


.0093 


51.14% 


3.14 


.0784 


Belief in/attitude 
toward pollution 
(df - 8, 162) 


.0057 


51.71% 


1.92 


.1673 


Perceived indiv. 
locus of control 
(df - 9, 161) 


.0036 


52.07% 


1.25 


.2649 


Perceived group 


.0008 


52.15% 


0.24 


.6261 



locus of control 
(df= 10, 160) 



Sia’s study w'as replicated by Sivek (1989; Sivek 
and Hungerford 1989/90), and replicated and extended 
by Marcinkowski (1989). Both studies corroborated the 
importance of Sia’s top three variables as predictors of 
REB. Of particular relevance to this issue and action in- 
struction program are the results from all three studies 
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Table 3. Summary of Stepwise Regression Results 
Regarding the Relative Ordering and Contribution to 
Explained Variance in REB for Knowledge of Action 
and Skill in Using Action 





Know, of Action 


Skill in Action 




Entry 


R-square 


Entry 


R-square 


Study Samples 


Order 


Value 


Order 


Value 


Sia Total 


3rd 


1 .8% 


1st 


4.5% 


(1985) Sierra Club 


3rd 


1.1% 


1st 


30.1% 


Sivek Total 


6th 


.03% 


1st 


18.6% 


(1989) WI Trapper's Assoc. 7th 


.02% 


1st 


28.1% 


Ducks Unlimited 


5th 


1.21% 


1st 


15.4% 


Trout Unlimited 


4th 


2. 1 2% 


1st 


11.1% 


Marcinkowski Total 


1st 


19.3% 4th 


5.3% 


(1989) 










pertaining to knowledge 


of and 


skill in 


using 


; action 



strategies. These results were summarized by Marcin- 
kowski (1998, p. 249). and are presented in Table 3. In 
ail three studies, and for all samples (separate and com- 
bined), one of these two variables served as the strongest 
predictor of REB. This either/or finding reflects the fact 
that the relationship between these two variables for 
Sia's sample (1985), Sivek’s (1989) samples and sub- 
samples, and Marcinkowski's (1989) sample were statis- 



tically significant in all cases (i.e., ranging from a low of 
r-.72 for Sivek's Trout Unlimited sample, to a high of 
r-.87 for Sivek’s Wisconsin Trapper’s Association sam- 
ple). 

Lierman (1995) carried out one of the few studies of 
predictors of REB using a sample of high school stu- 
dents. Her data (n-436) had been collected during the pi- 
lot test of the Secondary School Environmental Literacy 
Instrument (SSEL1; Marcinkowski and Rehrig 1995, 
Wilke 1995). The SSELI included comparable measures 
for Sia’s top five predictor variables and for REB. From 
Lierman’s stepwise regression analysis of the full range 
of measured variables, she found skill in using environ- 
mental action strategies to be the strongest predictor of 
REB for these students, followed by willingness to act 
(as in Hines 1985) and issue identification skills (as in 
Volk 1980). These results provide evidence that there is 
an important link between variables found to serve as 
correlates and predictors of REB and the variables that 
are featured as aims of this program. 

I-Iungerford and Volk (1990) conducted the third of 
the aforementioned research reviews as they prepared an 
invited paper. In this case, Hungerford and Volk did not 
collect their own data or rely on their own statistical 
analyses. Rather, they relied on prior reviews of research 
on REB. From their careful analysis of this literature, 
they generated a model of variables that w'ere related to 
behavior (Table 4). 



Table 4. Variables included in Hungerford and Volk’s (1990) Model 



Stase 


Maior Variables 


Minor Variables 


A. Entry 


Environmental sensitivity 


Know ledge of ecology 
Androgyny (psychological gender role) 
Attitudes toward pollution, 
technology, and economics 


B. Ownership 


In-depth knowledge 
of issues 

Personal Investment 

in issues and environment 


Knowledge of consequences of 
behavior (+ and -) 

Personal commitment to issue 
resolution 


C. Empowerment 


Knowledge of and 
skill in using 
action strategies 
Locus of Control 
Intention to act 


In-depth knowledge of issues 
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For the purposes of this paper, three things are notewor- 
thy about the variables included in Hungerford and 
Volk's model (Table 4): 

1. The model contains all major variables identified in 

Hines' (1985) meta-analysis, as well as in prediction 
studies by Sia (1985), Sivek (1989), Marcinkowski 
(1989) and Lierman (1995). These appear to be sta- 
ble and significant attributes of adults active in con- 
servation and/or environmental pursuits and, to a 
lesser extent, of adolescents; 

2. Most of these variables are found in modern environ- 

mental literacy frameworks (Simmons 1995, Wilke 
1995), and therefore also serve as aims of EE ("Pur- 
poses" in Figure i): and 

3. Most of these variables are emphasized in this issue 

and action instruction program, and evaluation stud- 
ies have found that they are influenced to a signifi- 
cant degree by this program. 



The model summarized in Table 4 serves as the core 
of the program theory for this issue and action program, 
and represents a significant departure from the popular 
but ineffective KAB model. The extent to which these 
correlates and predictors of REB can be influenced by 
this issue and action instruction program will be summa- 
rized in the next section. 



Evaluations of This Issue and Action Instruction Pro- 
gram 

As presented in Figure 1 and as discussed earlier, 
"Outcomes" is a central element of any logic model. Mi- 
chael Quinn Patton suggests that without evidence of 
program's outcomes, a program is often left to justify its 
worth using arguments based on intrinsic values (e.g., 
tradition, popularity, etc.). In this section, I will summa- 
rize the empirical basis for the "Outcomes" of this pro- 
gram, and their relationship to "Purposes," "Tools," and 
"Delivery Strategies." 



Table 5 - A Summary of Results for Experimental-Type Evaluation Studies of This Program 











Results for Program Aims as Outcome Variables * 










Env. 


Know. Know. 


Skill/ 


Know. 


Skill/ 


Locus of 










Sens. 


Ecol. Issues 


Issues 


Action 


Action 


Control 


REB 


Author. Date 


Grades 


Currie. 








A 


p 




In. 


Gr. 




Ramsey, 1979 


7th 


IEEIA 








X 










X 


Klingler, 1980 


8th 


IEEIA 








X 










X 


Volk, 1980 


8th 


IEEIA 




X 
















Ramsey, 1989 


7th 


IEEIA 


ns 






X 


X 


X 


X 


X 


X 


Ramsey, 1993 


8th 


IEEIA 


ns 






X 


X 


X 


ns 


X 


X 


Holt, 1988 


8th 


IEEIA 


ns 






X 


X 


X 


X 


ns 


X 


Bluhm et al. 1995 


6th 


IEEIA 










X 


X 






X 


Bluhm Sc McBeth, 1996 


6th 


IEEIA 










X 


X 






X 


Withrow, 1988 


5th/6th 


Case St. 




X 




X 


X 


X 


ns 


ns 


X 


Simpson, 1991 


5th/6th 


Case St. 








X 




ns 


ns 


ns 


X 


Culen, 1996 


7th/8th 


Case St 




X 




X 




X 


X 


ns 


X 


Jordan, 1986 


Noil-formal - 








X 










X 



Under Know. Action, A « Actual, and P - Perceived 

Under Locus of Control, In. = Individual, and Gr. = Group 

X = significant at p < .05: ns = not statistically significant; blank = not measured 
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There are several ways of collecting evidence about 
a program's outcomes. These include: (a) periodic as- 
sessment by teachers (e.g., using surveys); (b) periodic 
assessments of students (i.e., as in standardized testing); 
(c) evaluation studies using quantitative designs (e.g., 
experimental-type studies); and (d) evaluation studies 
using qualitative designs. The developers have used all 
four of these methods to determine the effects and effec- 
tiveness of this program's curricula and teacher prepara- 
tion efforts. 

Results from Experimental-Type Evaluation Studies 

This issue and action instruction program is unique 
in the fields of ARE and £E in that program-related 
evaluations have been conducted on a regular basis since 
its first year, 1973, and that many of these evaluations 
have been experimental in nature. Since the late 1970s, 
at least 12 experimental-type evaluation studies have ex- 
plored the effects and effectiveness of issue and action 
curricula in school classrooms and non-formal settings. 
Many of these have been reviewed in earlier research 
documents (Hines 1985; Hungerford 1988; Volk and 
McBeth 1997; Hungerford, Volk and Ramsey 2000). 

The benefit of experimental-type studies such as 
these is that they can provide sufficient control over in- 
ternal validity threats to permit cause -and -effect claims 
to be made with a reasonable degree of certainty or 
probability (Campbell and Stanley 1963). It is notewor- 
thy that several of these have been replication-type stud- 
ies to determine if results obtained in an earlier study 
were or were not due to this program. As stated earlier, 
the ability to support cause-and-effect claims is central 
to program impact theory and logic modeling, making 
experimental-type studies and replications of them par- 
ticularly valuable. 

Experimental-type Studies of IEEIA 

These 12 experimental studies have been organized 
into three groups: classroom studies of IEEIA, classroom 
studies of extended case studies, and one study con- 
ducted for a non- formal program (Table 5). The largest 
number of quasi -experimental studies have looked at the 
effects of IEEIA, largely because this is by far the oldest 
and most widely used of available issue and action cur- 
ricula. In the First of these studies, Ramsey (1979) com- 
pared lEEIA-based instruction with a ware ness -level case 
study instruction and with standard science instruction at 
the 7th grade level. He reported positive results on his 
two outcome variables, knowledge of action strategies 
and level of involvement in REB. In 1981, Ramsey 
oversaw a follow-up study of students who had been in- 
volved in liis 1979 study. Graduate students were trained 



to conduct double-blind interviews of students from both 
treatment and control classes. He found that these gradu- 
ate students were able to pick out every student from liis 
treatment group on the basis of interview responses 
alone. Further, he found that students exposed to IEEIA 
continued to engage in higher levels of REB than did 
their control group peers despite the absence of any in- 
structional reinforcement (i.e., with only social rein- 
forcement from family, peers, and youth groups) (Ram- 
sey 1981, Hungerford and Volk 1990). 

Kiingler’s study (1980) was undertaken as a replica- 
tion of Ramsey’s earlier study (1979). He used the same 
material, similar methods, and the same outcome meas- 
ures, but with 8th grade students. The consistency be- 
tween his results and Ramsey's results lends credibility 
to Ramsey's original findings (Table 5). 

Ramsey’s later studies (Ramsey 1989, 1993; Ram- 
sey and Hungerford 1989) involved seventh and eighth 
grade teachers who had completed an extended issue and 
action instruction in-service workshop, and who were 
implementing IEEIA in their own classrooms. These 
seventh and eighth grade classes were compared to other 
classes from those same schools that were exposed to 
standard science instruction. These studies may be seen 
as replications and extensions of the earlier studies and 
of each other, as well as evaluations of the efficacy of 
the previously described issue and action teacher prepa- 
ration programs, in these studies, Ramsey drew upon 
prior research (Ramsey 1979, Peterson 1982, Sia 1985) 
in developing his outcome measures. In both studies, he 
used five measures: (a) a Likert-type measure of indi- 
vidual and group locus of control; (b) a phenomenologi- 
cal measure of subjects’ overt REB; (c) a Likert -type 
measure of environmental sensitivity; (d) a phenomenol- 
ogical measure of students’ knowledge of environmental 
action strategies; and (e) a Likert-type measure of per- 
ceived knowledge or and skill in using environmental 
action strategies. In both studies, analyses of data from 
pre-test measures of variables (a) and (b) indicated that 
there were no significant differences between treatment 
and control groups (Ramsey & Hungerford, 1989, p, 33; 
Ramsey, 1993, p. 34). Further, as reported in Table 5), 
Ramsey found in both studies that the classes exposed to 
issue and action instruction scored significantly higher 
oil posttest measures of variables likely to be influenced 
by issue and action instruction than did the control 
classes (i.e., with the exception of Individual Locus of 
Control in the 1993 study). Finally, in both studies, 
Ramsey reported that treatment/control difference on 
post measures of environmental sensitivity were not sig- 
nificant. He inferred tliat while periodic outdoor experi- 
ences were prone to influence this variable, classroom- 
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based issue and action training was not (Ramsey and 
Hungerford 1989, p. 32; Ramsey 1993, p. 36). 

Holt’s study (1988) was conducted shortly after the 
data for Ramsey’s 1989 and 1993 studies had been col- 
lected (i.e., 1987). Her students were exposed to the 
same material, similar teaching methods, and the same 
five outcomes measured used in Ramsey’s 1989 and 
1993 studies. However, her study is unique in that the 
students exposed to IEEIA were lower achieving stu- 
dents. Her results indicate that these issue and action 
curricula can work as well with non-gifted as with gifted 
students (Table 5). 

Two additional quasi-experimenta! studies of the ef- 
fects of IEEIA were undertaken in the mid-1990s. In the 
first (Bluhm, Hungerford, McBeth and Volk 1995), the 
developers and several colleagues sought to refine and 
validate their Middle School Environmental Literacy In- 
strument (MSELI), a more comprehensive measure of 
learning outcomes associated with environmental liter- 
acy. The second study was undertaken as part of this 
MSELI instrument development effort, and was included 
in a report on IEEIA to the U.S. Department of Educa- 
tion's National Diffusion Network (Bluhm and McBeth 
1996). In both studies, 6th grades exposed to IEEIA 
(treatment group) were compared to 6th grades exposed 
to standard instruction (control group). Further both 
studies used a post-only design and found that the treat- 
ment classes outscored the control classes on all three 
outcome variables: (a) perceived knowledge of environ- 
mental action strategies; (b) skill in using environmental 
action strategies, and (c) seif-reported involvement in 
REB (Hungerford, Volk and Ramsey 2000). 

The remaining quasi-experi mental study of the ef- 
fects of IEEIA (Table 5) w'as conducted by Volk (1981; 
Volk, Hungerford and Tomera 1981). In it, Volk specifi- 
cally investigated the effects of issue and action instruc- 
tion on eighth-grade students' ability to identify and ana- 
lyze issues. She reported that students exposed to IEEIA 
were able name more issues, identify more sides to 
named issues, and provide more rationales for differing 
sides than did her control sample (p<01). This study 
provided the evidence to support the addition of the 
evolving approach to issue analysis to the 1985 edition 
of IEEIA (Ramsey, Hungerford and Volk 1989). 

Experimental-type studies of intermediate-level ex- 
tended case studies. Several of the studies describe 
above demonstrated that IEEIA had a more substantial 
and significant effect on knowledge, skills, and REB 
than did awareness-level case studies (Ramsey 1979; 
Childers 1981; Jordan, Hungerford and Tomera 1986). 
As described earlier, the developers of IEEIA prepared 
an extended case study version of their issue and action 



curriculum to overcome the apparent weaknesses of 
awareness-level case studies, as well as to provide a 
more appropriate way of providing issue and action in- 
struction to students in intermediate grades. Three quasi- 
experimental studies were undertaken to determine if the 
intermediate-level extended case study format would 
lead to the same learning outcomes as were apparent in 
studies of IEEIA (Withrow 1988, Simpson 1991, Culen 
1996). 

In the first of these research studies, Withrow 
(1988) explored the effect of an unpublished extended 
case study of his own design (i.e., on deforestation in the 
Midwest) on fifth and sixth grades using a post-only 
quasi-experi mental design. He reported that the class ex- 
posed to the extended case study scored significantly 
higher than did the class exposed to standard instruction 
(control class) on five of his seven outcome measures: 
(a) actual knowledge of environmental issues; (b and c) 
actual and perceived knowledge of environmental action 
strategies; (d) skill in using environmental action strate- 
gies, and (e) self-reported involvement in REB (Withrow 
1988; Hungerford, Volk and Ramsey 2000). He did not 
find significant differences on his measure of (f and g) 
individual and group locus of control. 

Simpson conducted a similar study involving 15 
classrooms of fifth and sixth grades from Illinois and 
Tennessee. She used an extended case study on Canada 
geese that she had helped to develop (Culen, Simpson, 
Hungerford, Ramsey and Volk 1988), and collected data 
using a post-only control group quasi-experimenta] de- 
sign. She measured the same outcome variables as 
Withrow' except for (a) knowledge of issues. On her 
measured outcome variables, her results were similar to 
Withrow' wath one exception. She did not detect a sig- 
nificant difference between treatment (IEEIA) and con- 
trol classes on (d) their skill in using action strategies 
(Table 5). 

In the most recent of these studies. Culen (1996, 
Culen and Volk 2000) explored the effects of an ex- 
tended case study that he had developed on wetlands 
(Culen 1992). His sample consisted of 15 intact classes 
of seventh and eighth graders from Illinois and Missouri. 
His design was similar to Ramsey’s (1979) original 
study, comparing lEEIA-based instruction w'ith aw'are- 
ness-leve! case study instruction and with standard sci- 
ence instruction (as a control). He used six outcome 
measures: (a) ecological foundations; (b) actual knowl- 
edge of environmental action strategies; (c) skill in using 
environmental action strategies; (d and e) individual and 
group locus of control; and (f) self-reported involvement 
in REB (Culen 1996; Culen and Volk, 2000; Hunger- 
ford, Volk and Ramsey 2000). He found significant dif- 
ferences on his three pre-test measures (i.e., a, d, and f), 
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and therefore used analysis of covariance (ANCOVA) to 
carry out posttest comparisons. He found that his IEEIA- 
based group significantly outscored the control group on 
all but one of these variables (i.e., e above; p<.05). He 
also reported that his IEE LA-based group significantly 
outscored his awareness-level case study group on the 
posttest measure of behavior (i.e., f, above). 



Experiment-type Studies in a Non-formal Setting 

At the time Jordan began his study, only one other 
study appearing in The Journal of Environmental Educa- 
tion had investigated the effects of a non-fonnal program 
on REB (i.e. Asch and Shore 1975). Results from sur- 
veys of program during this period suggested that few 
non-formal programs undertook any form of outcome 
evaluation (Disinger 1981, Chenery and Hammerman 
1 984/85), Thus, the study undertaken by Jordan was, and 
remains today, unique in light of these conditions within 
the non-formal sector. 

Working with six-day residential programs for high 
school students offered by an environmental center, he 
set up and compared an awareness-level workshop (i.e. 
focusing on natural history, ecology, environmental is- 
sues, and resource management) with issue and action 
workshops (i.e., focusing on environmental issues, re- 
source management, and environmental action strate- 
gies) (Jordan, 1985; Jordan, Hungerford and Tomera 

1985) . Material for the latter workshop was derived, in 
part, from the action paradigm appearing in IEEIA, 
though did not include the skill development and appli- 
cation material found in earlier chapters of IEELA. Par- 
ticipants in all workshops underwent both pre- and post- 
tests using the same two instruments used by Ramsey 
(1979) and Klingler (1980). His measures of REB w'ere 
designed to cover the 2-month period prior to each 
workshop (i.e., administered during the pre-test) and the 
2-month period after each workshop (i.e., administered 
as a delayed post-test). 

Jordan (1985; Jordan, Hungerford and Tomera, 

1986) reported that participants in the issue and action 
workshops demonstrated a significant pre-post gain in 
knowledge of action strategies (p<.05), as well as higher 
level of knowledge of action strategies on the posttest 
measure than did those in awareness-level workshops. 
The same results were reported for his measures of self- 
reported REB. Thus, with some modification and in the 
hands of trained instructors, non-formal programs can be 
designed and delivered to help learners more than 
awareness of problems/issues. 



Measurement of Outcome Variables in Experimental- 
Type Studies 

The instruments used in these studies to assess 
learning outcomes on a pre/post or post-test only basis 
evolved over time, as this program and its accompanying 
research and evaluation base evolved. The first instru- 
ment (Ramsey 1979; Klingler 1980; Ramsey, Hunger- 
ford and Tomera 1981; Jordan 1985) assessed only stu- 
dents' knowledge of environmental action and their level 
of REB. Even so, Ramsey remains one of the few to use 
other sources to corroborate self-reported REB data (i.e., 
to do so, he used a parental questionnaire on a blind ba- 
sis). His 1979 instrument is shorter and simpler than the 
multi-sectioned instrument used in Ramsey's later stud- 
ies (1989, 1993), and in studies by Holt (1988), Withrow 
(1988), Simpson (1991), and Culen (1996). 

Efforts by Ramsey, prior researchers, and other de- 
velopers helped to evolve and validate a multi-section 
instrument to assess environmental literacy and evaluate 
this program at the middle school level. This instrument 
came to be known as the Middle School Environmental 
Literacy Instrument (MSELI). By 1993, MSELI was in 
its fourth edition. Over 1994-95, MSELI received 
greater scrutiny as part of the "Environmental Education 
Literacy/Needs Assessment Project” (Bluhm, Hunger- 
ford, McBeth and Volk 1995). Following this project, 
MSELI was further refined and used to evaluate this 
program (e.g. Bluhm and McBeth 1996). The evolution 
of this instrument, along with supporting validity and re- 
liability estimates, was chronicled by McBeth (1998). In 
these ways, the developers of this issue and action pro- 
gram attempted to insure that the instrument used to as- 
sess learning outcomes were valid and reliable, as well 
as consistent with the program's research base. Jn the 
language of the logic models presented earlier (Figure 2 
and Figure 3), these efforts helped insure that "Partici- 
pant Outcomes” could be attributed to the "Tool Box,” 
and that these "Participant Outcomes" were conceptually 
and methodologically consistent with both "Program 
Purposes” and sought "Program Impacts." 

Results from Periodic Assessment of Students 

Prior to the widely distributed report, Closing the 
Achievement Gap (Lieberman and Hoody 1998). few ef- 
forts had been made to explore or evaluate the effects of 
EE programs in schools on subject area learning out- 
comes. Today, in the climate of national and state stan- 
dards, increased calls for accountability, and increasing 
use of high-stakes assessments to determine the worth of 
school programs, few question the need for EE to ex- 
plore or evaluate these effects. It is noteworthy tliat be- 
fore Lieberman began his project and study, and before 
these trends became prominent in K-12 schooling in the 
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U.S., Hungerford and his colleagues oversaw at least 
two such studies. Readers should note that these two 
studies are substantially different than the previously de- 
scribed classroom evaluation studies in that the outcome 
variables are measured using standardized tests. 

Both of these studies were initiated in 1992. In the 
first. Gavilla (1992) sought to compare issue and action 
instruction against more traditional environmental sci- 
ence instruction using the Iowa Test of Basic Skills 
(ITBS) commonly used in Kansas City area schools. 
Both classes received instruction over a semester, and 
were assessed using TTBS on a post-only basis. Gavilla’s 
analysis of scores from the Science Achievement and the 
Social Studies Achievement Tests indicated that the 
class exposed to issue and action instruction scored sig- 
nificantly higher than did the class exposed to environ- 
mental science instruction (p<.05). 

Hungerford and his colleagues at Southern Illinois 
University undertook the second study in 1992. They 
contacted and received standardized achievement test 
data from 28 teachers who had been trained in issue and 
action instruction. These teachers were from eight states 
(i.e., l'A. IL, KY, MD, MO, TN, TX, WI), and taught in 
grade levels 5 - 12. They reported "ninety-six percent 
(96%) of the respondents indicated that the achievement 
of students experiencing this (issue and action) approach 
was greater than that of peers who had not experienced 
this training" (Hungerford, Volk and Ramsey, 2000, p. 
8). Of these 28 teachers, only one indicated that 
achievement was only "the same or greater" when com- 
pared to standard subject area instruction. They go on to 
summarize these teachers’ results as follows: 

When questioned about the areas of 
achievement. 86% reported greater 
achievement in science, 46% reported 
greater achievement in social studies, 54% 
reported greater achievement in language 
arts, and 75% reported greater achievement 
in library skills. Sixty-one percent (61%) of 
the respondents reported greater overall 
achievement on the part of students experi- 
encing the issue investigation approach 

(2000. p. 8). 

In light of the positive results from both studies, 
Hungerford, Volk, and Ramsey inferred ’’that the acqui- 
sition of those liigher order skills (evident in issue and 
action curricula) permits students to function at a liigher 
order of cognition when approaching the standardized 
tests of achievement" (2000, p. 7). 



Results from Periodic Assessments by Teachers 

To date, there have been few attempts to survey 
teachers about their classroom experiences with, or 
about the apparent effects of, issue and action instruc- 
tion. The 1992 survey by Hungerford and his colleagues 
was presented in the previous section because teachers 
were reporting results from standardized achievement 
tests (i.e. as opposed to their own perceptions). Braun 
(1997) undertook the only other survey of teachers 
trained in issue and action instruction. 

There were two purposes to Braun’s study (1997). 
The first was to survey teachers who had completed at 
least one issue and action training workshop on a num- 
ber of independent variables, and four dependent vari- 
ables (i.e., perceptions of student achievement in four 
goal areas). The second purpose of this study was to de- 
termine the extent to which those independent variables 
helped explain variability (variance) in the dependent 
variables using multiple regression and correlation 
(MRC ) statistical methods. 

To gather these data, she developed a three -part 
measure: (Section 1) a measure of eight independent 
variables (e.g., teaching background, training in and use 
of issue and action curricula); (Section 2) a dependent 
measure consisting of 21 learning outcomes in four goal 
areas; and (Section 3) a measure of an independent vari- 
able, "outside influences on classroom implementation." 
Section 2 was based upon the Goals for Curriculum De- 
velopment in Environmental Education (Hungerford, 
Peyton and Wilke 1980b), prior research (Volk 1983), 
and material emphasized in issue and action curricula. 
Hungerford, Volk, and Ramsey reviewed a draft of this 
measure. The final version of it consisted of 21 learning 
outcomes measured using a five-point student achieve- 
ment scale (i.e., 0-Not at all, 2— Moderate, 4-Complete). 

Braun’s sample consisted of 360 teachers from 1 1 
states who had completed at least one issue and action 
instruction workshop, and whose names appeared in a 
database managed by faculty at Southern Illinois Uni- 
versity as part of their National Diffusion Network 
(NDN) project. Of these, 99 teachers returned responses, 
although only 65 of these were usable (i.e., from teach- 
ers who had implemented issue and action curricula). 
With respect to her first purpose, these 65 teachers 
tended to teach science (n-48). teach in grades 7-8 
(n-31), and had completed at least one graduate degree 
(n-37). On average, they completed two training work- 
shops (mean-1.99), lasting more than two weeks (mean 
-2.3), and had implemented issue and action instruction 
for almost two years (mean - 1.97). The majority of 
these teachers reported using 1EE1A or 1ESTS1S (n~39), 
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while smaller numbers reported using extended case 
studies (n-1 1), or a combination of the two (n-13). 

With respect to the dependent variables, teachers' 
perceptions of student acliievement were highest for 
learning outcomes under Goal Level 2: Issue Awareness 
(mean -2.55). This followed by outcomes under Goal 
Level 3: Issue investigation/evaluation (mean -2.37) and 
under Goal Level 1: Ecological Foundations (mean 
=2.28). The latter can be partially explained by the fact 
that majority of teachers reported using IEEIA/IESTSIS, 
neither of which were designed to address Goal Level 1 . 
The teachers’ lowest perceptions of student achievement 
were for learning outcomes under Goal Level 4: Citizen 
action skills (mean =1.80, or below ’’Moderate"). It is 
noteworthy that these teacher perceptions pertain to 
achievement of learning outcomes related to action- 
related knowledge and skills. This may help to explain 
their apparent inconsistency with the positive results on 
REB, or action-related behavior, reported in all of the 
classroom evaluation studies (see Table 5). Two other 
plausible explanations for the apparent inconsistency be- 
tween these ratings and prior results are (a) satdents may 
engage in a higher level of REB through exposure to this 
program as a whole, even when teachers and students do 
not address all elements of action training in these cur- 
ricula, and (b) teachers’ perceptions of student learning 
outcomes associated with Goal Level 4 may not ade- 
quately reflect students’ out-of-school involvement in 
REB, and therefore may be limited. 

The results of Braun's MRC analyses are interest- 
ing. She reported that the list of 1 1 "Outside influences’’ 
was the only independent variable found to serve as a 
significant predictor of student achievement for each of 
the four goal levels (p<.05). Further analyses were not 
carried out to determine which of these eleven may be 
influence teachers' perceptions of student achievement at 
each goal level (e.g., administrative support, prior 
knowledge of ecology and environmental science, fit to 
their school's curriculum). Only two other independent 
variables were found to serve as significant first-order 
predictors of teachers' perceptions of student achieve- 
ment: for Goal Level 1: Ecological foundations, the 
grade level at which they taught; and for Goal Level 2; 
Issue awareness, the subject area in which they taught. 
In light of teachers' perceptions and these results, Braun 
called for research of both a quantitative and qualitative 
nature. 

Results from u Qualitative Evaluation Study 

At least within the U.S., limited attention has been 
given to qualitative evaluation in the fields of ARE and 
EE. This is also true for tliis issue and action instruction 
program. For tliis program, the low level of attention to 



qualitative evaluation parallels the level of attention 
given to periodic assessments of students and to assess- 
ments by teachers (i.e., in comparison to the substantial 
attention that has been given to experimental-type class- 
room evaluation studies). There are mitigating reasons 
for this. From an historical perspective, many of the ex- 
perimental-type evaluation studies of tliis program were 
carried out prior to the rise of qualitative methods in EE 
in the 1990s (Mrazek, 1993; see Table 5). Further, from 
a practical perspective, the amount of training, prepara- 
tion, time, and effort required to carry out sound qualita- 
tive evaluation studies has often limited their use. Fi- 
nally, from the developers' perspective and the perspec- 
tive of logic modeling, experimental-type studies pro- 
vide evidence regarding cause-and-effect claims, and 
therefore are of unique value to this program. Given 
these factors, it is noteworthy that Winther (1999) de- 
cided to carry out a qualitative evaluation of this issue 
and action program for his dissertation. 

In tliis study, Winther (1999) explored teacher deci- 
sion-making during the first year of implementation of 
IEEIA (i.e., following their completion of a summer 
training workshop). From the teachers who completed 
this workshop and who volunteered to participate in this 
study, he randomly selected eight teachers: three high 
school, two middle school, and three elementary teach- 
ers. Selected teachers were from different states (e.g., 
MD, MO, TX). 

His final four questions focused on: (1) what moti- 
vated teachers to participate in the training for and to 
implement IEEIA: (2) teachers’ perceptions of this train- 
ing; (3) how teachers working with IEEIA perceive the 
support they receive from peers and administrators; and 
(4) the perceptions and reactions of teachers as they im- 
plement (or interact) with IEEIA. While this study was 
not designed to do so. these questions clearly explored 
variables or factors included in Braun's (1997) list of 
"Outside Influences" that were found to predict teachers' 
perceptions of student achievement in this program (e.g., 
prior knowledge of ecology' and environmental science, 
fit of issue instruction to their curriculum, adequacy of 
preparation, support from administration). 

To gather information and perceptions needed to re- 
spond to these research questions, he interviewed each 
teacher three times: at the close of the summer workshop 
prior to implementation, at midyear during implementa- 
tion, and at the end of the spring term toward the end of 
implementation. Other sources used to verify these 
teachers’ responses included visits to their schools, ex- 
amination of teacher- and student-prepared materials, in- 
terview's with their students, informal discussions with 
peer teachers at their school, and implementation reports 
they submitted. Winther’s qualitative methods reflect a 
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naturalistic approach, inductive process, design flexibil- 
ity, and personal contact, as well as use prolonged en- 
gagement and triangulation to help establish the depend- 
ability and credibility of information and perceptions 
(Guba 1981, Patton 1990). 

In response to the first two research questions 
above. Wiiither concluded that: 

Participants initially found the training diffi- 
cult and confusing. As they continued train- 
ing and implementation some participants 
found that the program was substantially 
different from what they were accustomed 
to doing in their classrooms. Some partici- 
pants were reluctant to implement the pro- 
gram ... Others implemented the program 
fully, and were extremely enthusiastic about 
what their students were doing and learning 
(1999, Abstract). 

With respect to the third research question noted above, 
Winther reported that support from administrators and 
peers was important, but not as important as previously 
thought. 

Administrators w'ere perceived as supporting 
the program [IEEIAJ because the program 
helped facilitate goals that had largely been 
established in the school. Peer support was 
social and largely passive. Participants per- 
ceived that neither administrators nor peers 
really understood how the program worked 
(1999. Abstract). 

Winther concluded that teachers’ perceptions of the 
training, the program, and support base stemmed from 
what he referred to as "cognitive changes” srimulated by 
the training and program. "Comments from all of the 
teachers reflected reactions to these changes. Some 
teachers resisted the changes, while others enjoyed the 
process" (1999, Abstract). 

Winther’s study confirms what the developers, 
trainers, and researchers associated with this program 
have learned tlirougii experience; i.e., some teachers are 
more "ready” than others to change the way in which 
they perceive the teaching and learning process, and 
their own roles in that process. To effectively use this is- 
sue and action program, teachers must be willing to 
change and grow. For this reason, Winther (1999) has 
recommended that factors associated with "teacher deci- 
sion-making" (i.e., this change process), be explored in 
greater depth. From a practical perspective, research of 
that kind could help guide workshop advertising, re- 
cruitment, and selection processes. From the perspective 



of logic modeling, research of that kind could be used to 
insure that the "Delivery Approaches" for this program’s 
"Tools" and methods to "Support Tool Use" were appro- 
priate for its intended "Target Audiences." 



Recommendations 

For the 28 years over which this issue and action 
program has evolved, much has been learned that is 
relevant to the goals of Stewardship Education, be that in 
the field of ARE or EE. Its developers have learned 
much about designing and developing skill-based and 
project-driven curricula such as IEEIA and extended 
case studies, and about how to prepare teachers to effec- 
tively use these curricula. They have carried out research 
studies and supported the synthesis of research in an ef- 
fort to use empirical findings to help guide the evolution 
of these curricula and their teacher preparation pro- 
grams. They have also carried out various kinds of 
evaluations to determine how well these curricula and 
teacher preparation programs are working, and to further 
refine them. This systematic use of data, coupled with 
attention to what has worked in practice, has made this 
issue and action program sound, credible, and somewhat 
unique in ARE and EE. While this issue and action pro- 
gram is not as widely used across the U.S. as some other 
ARE or EE programs, the logic models that under-gird 
this program, as described in this paper, are substantial. 
For that reason, its key elements deserve careful scrutiny 
and consideration by any professionals who work in the 
area of Stewardship Education. On the basis of the ac- 
complishments (to date) within this issue and action pro- 
gram, the following recommendations are offered to pro- 
fessionals working in (he area of Stewardship Education. 

Purposes 

1. The organization’s mission, overall educational aims, 
more specific education program goals, and instruc- 
tional objectives should be aligned w'ith one another 
so that they all clearly reflect the nature and purpose 
of Stewardship Education: To prepare learners to be 
actively involved in the stewardship of ecological 
and natural resources. 

2. The organization's educational aims should reflect 
the best of w'hat is known about what it takes for 
citizens to become actively and effectively involved 
in stewardship efforts on a sustained basis. In this 
paper, I Iiave suggested that these aims are apparent 
in evolving, research-based models of environ- 
mental literacy (Simmons 1995, Wilke 1995. Volk 
and McBeth 1997). 
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3. The organization's educational goals and objectives 

should be structured and sequenced in such a way 
that they encompass and foster what is known about 
environmental literacy (aims). 

(a) These goals and objectives should not be limited 

to awareness, knowledge, and/or attitudes if 
stewardsliip is a desired outcome. These goals 
and objectives must also address the other af- 
fective dispositions (e.g., locus of control, as- 
sumption of personal responsibility), skills, and 
participation strategies that are more closely re- 
lated to learner (and citizen) participation in 
stewardsliip. 

(b) Further, these goals and objectives should not 
focus solely upon the natural world, or even 
upon problems and issues if stewardship is a 
sought program impact. These goals and objec- 
tives must focus, at least in part, on the kinds of 
solutions and service/action projects that make 
stewardsliip a concrete reality. In this paper, I 
have indicated that the developers of this pro- 
gram have used and adapted the Goals for Cur- 
riculum Development in Environmental Educa- 
tion (Hungerford, Peyton and Wilke 1980b) as 
their guide for program goals and objectives. 

4. Program goals and objectives should be viewed as 
anticipated learning outcomes, and compared to a 
program’s actual learning outcomes as determined 
through assessment and evaluation efforts. This in- 
volves linking Program "Purposes" to "Outcomes" 
and "Impacts" on an empirical basis. 

Curricula 

5. Curricula for stewardship education should be struc- 

tured in such a way that learners will be provided 
with a well-thought-out sequence of opportunities to 
develop, build upon or practice, and eventually ap- 
ply their awareness, knowledge, affective disposi- 
tions, skills, and participation strategies. To ac- 
complish this, the developers of this program have 
relied on the Goals for Curriculum Development in 
Environmental Education (Hungerford, Peyton, Sc 
Wilke, 1980b), as well as their knowledge of skill 
development and application methods, and methods 
for overseeing student projects. 

6. Stewardship-oriented curricula that are project- 
driven should be structured so that learners are ade- 
quately prepared to undertake each step of that pro- 
ject. Further, they should be structured so that 
teachers can prepare learners for (as instructors) and 
guide learners tlrrough (as facilitators) the entire 
project process. 



7. These opportunities for learners to develop, build 
upon or practice, and eventually apply their aware- 
ness, knowledge, affective dispositions, skills, and 
participation strategies should take into considera- 
tion the learners' developmental level(s), prior 
knowledge and experience, and aptitude (i.e., these 
opportunities should be challenging, but within their 
reach). To reflect these considerations, 1EE1A was 
designed for use with middle level students, while 
many of the earlier extended case studies w'ere de- 
signed for younger learners (see differences in "De- 
livery Strategies" in Figures 2 and 3). 

8. Curricula should be periodically reviewed and, as 
appropriate revised to reflect recent feedback from 
teachers and learners, results of recent assessment 
and evaluation studies, results from recent research 
studies and syntheses, and other sources (e.g., na- 
tional and state standards; parents). The earlier nar- 
rative on the evolution of IEE1A is a good example 
of how this has been done within this issue and ac- 
tion program. 

Teacher Preparation 

9. WTien preparing teachers to work with stewardship- 
oriented programs and curricula, adequate time 
must be provided for teachers to develop, build 
upon or practice, and eventually apply their own 
awareness, knowledge, affective dispositions, skills, 
and participation strategies. One-shot workshops 
will never be able to accomplish this. If prolonged 
in-service workshops such as those used with this 
issue and action program, are not available, other 
options should be considered (e.g.. a series of work- 
shops for the same teachers). 

10. Teacher preparation programs should help teachers 
become intimately familiar with curricula and other 
resources they will be asked or expected to use in 
their own classrooms. This is why ail graduate 
courses and in-service workshops for this program 
emphasize the use of program curricula. 

1 1. When preparing teachers to use stewardship-oriented 
curricula that are project-driven, teachers must be 
provided with an opportunity to engage in such pro- 
jects themselves, even if on a somewhat simplified 
basis. As has been the experience in this issue and 
action program, only when teachers engage in pro- 
jects themselves do they become more fully aware 
of project requirements, components, procedures, 
difficulties, and associated evaluation and grading 
procedures. 
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12. When preparing teachers to use stewardship-oriented 
curricula, several other features will often increase 
teachers’ effective use of them, as lias been apparent 
in tills issue and action program: 

(a) Providing these teachers with adequate support 

as they plan and implement that curriculum. 
This should include instructional, resource, and 
assessment support; 

(b) Providing these teachers with opportunities to 
share with other trained teachers their questions 
and efforts related to curriculum planning, im- 
plementation, and assessment and evaluation; 
and 

(c) Providing these teachers with subsequent oppor- 

tunities to refresh and expand upon what they 
got out of initial preparation opportunities (e.g., 
extended workshops). 

Assessment and Evaluation 

13. Evaluations of ARE and EE programs must move 
beyond the collection of program outputs (i.e., 
number counts and feedback; see Chenery AMD 
Hammerman 1984/85) to include the assessment of 
learning outcomes and, whenever feasible, longer- 
term program impacts. In this issue and action pro- 
gram, assessment of learning outcomes has been 
viewed and undertaken in this manner since 1973, 
particularly in the previously described classroom 
evaluation studies. In some studies, program im- 
pacts have also been assessed (e.g., Ramsey 1981). 

14. In many instances, the assessment of learning out- 
comes will be undertaken for evaluation purposes 
(e.g., in most classrooms). When appropriate, the 
assessment of learning outcomes should include 
pre-assessment (e.g., on prerequisites, instructional 
objectives), formative assessments, and summative 
assessment. In some instances, the assessment of 
learning outcomes may be undertaken for research 
purposes (e.g., in classroom studies for theses and 
dissertations). In these cases, the use of pre-testing 
(i.e., a form of or a substitute for pre-assessment) 
and post-testing (i.e., a form of or substitute for 
summative assessment) should also reflect appro- 
priate research procedures and controls (e.g., as in 
Culen 1996). 

15. As recommended under the Purposes Section above, 
learning outcomes to be assessed should not be lim- 
ited to the main features of the KAB model (i.e.. 
awareness, knowledge, and attitudes). Rather, ef- 
forts should be made to assess a broader range of 
learning outcomes associated with environmental 
literacy and stewardship (e.g., the other affective 
dispositions such as locus of control, assumption of 



personal responsibility, skills, participation strate- 
gics). Assessment instruments such as the MSEL1 
used in this issue And action program (Bluhm, 
Hungerford, McBeth and Volk 1995; McBeth 
1997), and the related SSEL1 (Marcinkowski and 
Rehrig 1995) offer some ideas as to how this can be 
done. 

16. Regardless of whether the assessment of learning 
outcomes is undertaken using quantitative and/or 
qualitative methods, steps should be taken to insure 
that resulting assessment data are "trustworthy" 
(Guba 1981). In quantitative terms, this means that 
steps must be taken to insure that assessment in- 
struments yield valid and reliable data. In qualita- 
tive terms, this means that other methods must used 
to insure that the resulting assessment data are 
credible and dependable (Guba 1981). In this issue 
and action program, substantial attention has been 
given to instrument validity and reliability (Bluhm, 
Hungerford, McBeth and Volk 1995; McBeth 
1997), particularly when this has involved the col- 
lection of self-reported data (e.g., Ramsey 1979) 
and perception data (see Table 5 for studies meas- 
uring both actual and perceived knowledge of ac- 
tion strategies). Further, in Winther's (1999) quali- 
tative evaluation study, clear attention was given to 
methods recommended by Guba (1981) and others 
to insure credibility and dependability. 

17. The results of evaluation-oriented assessments of 
learning outcomes should be used by program staff 
as opportunities for the purposes of program learn- 
ing (i.e., how this program is working) and pro- 
gram improvement (i.e., w'here a program’s curricu- 
lar, instructional, and'or assessment practices need 
to be improved). Once a program is well defined 
and established, as with this issue and action pro- 
gram, research-oriented assessment results may be 
used for other purposes, including program valida- 
tion (Bluhm and McBeth 1996) and logic model- 
ing, as has been done in this paper. 

Research 

18. One of the most influential decisions to be made in 

any stewardsliip education program is whether or 
not to design, develop, implement, and as- 
sess/evaluate that program in light of prior research. 
In ARE and EE, many programs still rely on the in- 
effective KAB model described earlier, an indica- 
tion that inadequate attention is being paid to prior 
research. The evolving research base in EE. includ- 
ing the previously described research base for this 
issue and action program, provides strong cues as to 
what stewardship education programs should in- 
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elude and address. As a profession, we need to pay 
greater attention to what has been learned through 
research 

19. Once the above decision has been made, it is also 
important that professional educators stay current 
with this evolving body of research and participate 
in its evolution. For many in the fields of ARE and 
EE, both of these activities will require some addi- 
tional training to enable them to do so (e.g., in re- 
search methods, as critical consumers of research, 
as collaborators in research). The developers of this 
program and many of its trainers have been both 
beneficiaries of and provider of this through gradu- 
ate-level research training. 

20. From the perspective of logic modeling, many im- 
portant research questions can and should be gener- 
ated by asking about the relationships be- 
tween/among a program's elements. A list of gen- 
eral questions drawn from logic modeling prepared 
by Rossi, et al. (1999), was presented earlier in the 
paper. If needed, that list and the general logic 
model (Figure l) can serve as a starting point. Be- 
yond that, in the fields of ARE and EE, one of the 
most fundamental questions to be asked and ex- 



plored is this: "Are our goals and objectives, our 
tools, and our methods sufficient to help learners to 
attain learning outcomes that are critical to their ef- 
fective participation in stewardsliip activities?". 
This question has been given substantial attention 
by Hungerford and his colleagues, as is reflected in 
the research and the evaluation efforts that are 
summarized herein. 
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